The ubiquity of computing devices to daily life continues to grow. They take the form of personal and professional desktops, laptops, hybrid laptops, tablet PCs, e-book readers, mobile phones, smartphones, wearable computers, global positioning system (GPS) units, enterprise digital assistants (EDAs), personal digital assistants (PDAs), game consoles, and the like.
Computing devices generally consist of at least one processing element, such as a central processing unit (CPU), some form of memory, and input and output devices. The variety of computing devices and their subsequent uses necessitate a variety of input devices. One such input device is a touch sensitive surface such as a touch screen or touch pad wherein user input is received through contact between the user's finger or an instrument such as a pen or stylus and the touch sensitive surface. Another input device is an input surface that senses gestures made by a user above the input surface. Either of these methods of input can be used generally for drawing or inputting so-called digital ink to express text, symbols, etc., which the computing device interprets using handwriting recognition systems or methods. Other systems for handwriting input to computing devices include electronic or digital pens which interact with paper, encoded surfaces or digitizing surfaces in order to have their movement relative to the surface tracked by a computing device, such as the systems provided by Anoto AB., Leapfrog Enterprises, Inc., and Livescribe, Inc.
Regardless of the input method used, handwriting recognition systems and methods typically involve determining the initiation of a digital ink stroke, such as when first contact with a touch sensitive surface is made (pen-down event); the termination of the stroke, such as when contact with the touch sensitive surface is ceased (pen-up event); and any movement (gestures or strokes) made between stroke initiation and termination. These determined strokes are processed to interpret the input which is usually performed in several stages including preprocessing, segmentation, recognition, and interpretation. Generally, the preprocessing stage involves discarding irrelevant input data and normalizing, sampling, and removing noise from relevant data. The segmentation stage specifies the different ways to break down the input data into individual elements to be recognized depending on the type of input, e.g., characters, words, symbols, objects, or shapes. The recognition stage generally includes a feature extraction stage, which characterizes the different input segments, and a classification stage which associates the segments with possible recognition candidates. The interpretation stage generally involves identifying the elements associated with the candidates. Less, more, or different stages are also possible.
The type of computing device or input surface can also determine the type of handwriting recognition system or method utilized. For instance, if the input surface is large enough (such as a tablet), the user can handwrite input anywhere on or above the input surface, as if the user was writing on paper. This however adds complexity to the recognition task, because the separate elements to be recognized may be related dependent of the relative positions of the elements or may be unrelated independent of their relative positions.
For example, one desired use for handwriting recognition is in note-taking for the capture of mathematical equations or expressions, physics concepts, chemistry formulas, musical notation, etc., during education sessions, such as classes or lectures. That is, a student may wish to write multiple equations over several lines to express the working of a mathematical problem which the educator has demonstrated (which could also be in digital ink) or which the student is required to solve as an assignment or assessment, or the educator may wish to prepare a worksheet for students involving a list of non-related equations that define a set of problems to be solved by the student, either manually or automatically by the computing device, or the capture of a system of equations or vector/matrix may be desired. The need for the entry of multiple connected or un-connected expressions may also occur in enterprise settings, such as during budget setting meetings, technology research and development sessions, technical documentation, etc., or in personal settings, such as a consumer writing a long addition over several lines whilst grocery shopping in order to calculate the total amount.
Systems for the recognition of handwritten mathematical equations are known. These systems concentrate on determining the elements of input equations through matching against databases/lexicons containing known mathematical symbols and relationships. These systems generally recognize the elements without any consideration of the actual content or structure of the equations themselves. As such when multiple equations are entered, say in a vertical list, it is possible that the recognition may consider elements of intended separate equations to belong to the same equation, or at least the recognition element will form and test hypotheses with respect to this. This of course substantially increases recognition processing and time, and decreases recognition accuracy.
Some known systems relate to providing calculations or likely solutions of input equations, and therefore may take the content into account. However, these systems do not recognize multiple equations input either, rather they recognize the input of mathematical operators, such as the equals sign or a result line, or user gestures, to determine when a solution is to be provided to the currently input equation, such that the next input is inherently another separate equation or an edit to the current equation, see for example European Patent No. 0 676 065.
Other known systems provide recognition of systems of equations and tabular structures, such as matrices, involving equations. However, these systems rely on indicative elements for recognition, such as brackets or spatial alignment, e.g., within rows and columns, and as such do not recognize multiple equations as such rather a structure involving multiple inputs of any type, see for example U.S. Pat. No. 7,447,360.
What is required is a system that recognizes multiple equation inputs independent of links between the equations that do not rely on the input of specific designation elements or gestures and do not significantly increase processing time or complexity to the recognition of the equations themselves whilst retaining sufficient recognition accuracy.